Priority mechanism for distributed sending of media data

ABSTRACT

Media data is distributed across multiple devices, and multiple schedulers and senders are installed across the multiple devices. Schedulers formulate send requests that stipulate particular media data portions and designate destination clients. Each send request is associated with a send priority. The send requests are transmitted from the multiple schedulers to the multiple senders. The senders rank respective received send requests according to their associated send priorities. Based on the ranked respective send requests and responsive to a respective send bandwidth of each sender, the senders ascertain respective threshold priorities. Each respective threshold priority corresponds to the lowest send priority of all send requests that have been accepted for sending by a respective sender. By comparing individual send priorities associated with individual send requests to respective threshold priorities, the schedulers are able to determine which send requests are selected for sending by respective senders.

TECHNICAL FIELD

This disclosure relates in general to the distributed sending of mediadata and in particular, by way of example but not limitation, to apriority mechanism for the distributed sending of media data.

BACKGROUND

Television-based entertainment systems traditionally rely on cablenetworks in which the same television programming is broadcast to everyclient. In other words, the same television channels and pay-per-viewmovies are broadcast at the same time to every client that is connectedto the cable network.

This broadcasting approach with cable networks entails two drawbacks.First, because all programming content is broadcast at a single time, aviewer has no control regarding the time at which a program is viewed.Second, with all programs broadcast to every client, bandwidthlimitations render impracticable true video-on-demand (VOD) for justmovies alone, much less for television programming in general.

FIG. 1 illustrates an existing approach 101 to television programmingdissemination. Television programming dissemination approach 101 is anexisting attempt to counter the above-noted drawbacks. Multipletelevision programming sources 103(1), 103(2) . . . 103(j) store andprovide television programming. A central collector and forwarder 105determines when television programming needs to be sent to clients (notshown) through a network switch 107 via a network (also not shown).

Each request 109(1, 2 . . . j) for television programming is sent fromcentral collector and forwarder 105 to a respective televisionprogramming source 103(1, 2 . . . j). In response, each respectivetelevision programming source 103(1, 2 . . . j) provides requestedtelevision programming portions as collecting communications 111(1, 2 .. . j) to central collector and forwarder 105. Central collector andforwarder 105 thereafter forwards the television programming portions asforwarding communications 113(1, 2 . . . j) to network switch 107 andthence to the clients via some network.

Each requested television programming portion is therefore routedthrough central collector and forwarder 105, which can become abottleneck, prior to being sent to clients by way of network switch 107.Moreover, television programming dissemination approach 101 asillustrated in FIG. 1 is somewhat simplified. Although not so explicitlyillustrated, requested television programming portions as collectingcommunications 111(1, 2 . . . j) are actually provided to centralcollector and forwarder 105 from television programming sources 103(1, 2. . . j) by way of network switch 107.

Consequently, television programming portions that are destined forvarious clients traverse network switch 107 twice. Televisionprogramming portions are sent through switch 107 first to centralcollector and forwarder 105 as collecting communications 111(1, 2 . . .j) and then second from central collector and forwarder 105 asforwarding communications 113(1, 2 . . . j). The (i) central collectionand funneling and (ii) double switch traversal present additionallimiting factors. Hence, bandwidth and bottleneck issues continue to bepresent with television programming dissemination approach 101.

SUMMARY

Media data is distributed across multiple devices, and multipleschedulers and senders are installed across the multiple devices.Multiple respective senders are associated with multiple respectivedistributed portions of the media data. Multiple schedulers areassociated with clients and are capable of scheduling the sending ofmedia data thereto. Schedulers formulate send requests that stipulateparticular media data portions and designate destination clients. Eachsend request is associated with a send priority. The send requests aretransmitted from the multiple schedulers to the multiple senders. Thesenders rank respective received send requests according to theirassociated send priorities. Based on the ranked respective send requestsand responsive to a respective send bandwidth of each sender, thesenders ascertain respective threshold priorities. Each respectivethreshold priority corresponds to the lowest send priority of all sendrequests that have been accepted for sending by a respective sender. Thethreshold priorities are transmitted from the multiple senders to themultiple schedulers. By comparing individual send priorities associatedwith individual send requests to respective threshold priorities, theschedulers are able to determine which send requests are selected forsending by respective senders.

Other method, system, approach, apparatus, server, device, media,procedure, arrangement, etc. implementations are described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The same numbers are used throughout the drawings to reference likeand/or corresponding aspects, features, and components.

FIG. 1 illustrates an existing approach to television programmingdissemination.

FIG. 2 is an exemplary environment for media data disseminationarchitecture.

FIG. 3 illustrates exemplary media data dissemination architecture thatincludes devices for distributed sending of media data.

FIG. 4 illustrates exemplary devices for distributed sending of mediadata that is striped across the devices.

FIG. 5 illustrates an exemplary approach to distributed media datadissemination that includes messages used by multiple devices.

FIG. 6 illustrates a first aspect of an exemplary approach to a prioritymechanism for distributed sending of media data.

FIG. 7 illustrates a second aspect of the exemplary approach to apriority mechanism for distributed sending of media data.

FIG. 8 is a flow diagram that illustrates an exemplary method for apriority mechanism for distributed sending of media data.

FIG. 9 illustrates an exemplary send negotiation per-time-slot thatinvolves a priority mechanism for distributed sending of media data.

FIG. 10 illustrates an exemplary logical organization of media data.

FIG. 11 illustrates an exemplary approach to physically locating mediadata that includes a hash table.

FIG. 12 illustrates an exemplary approach to a popularity determinationfor media data.

FIG. 13 illustrates an exemplary hash table for locating media data,including replicated media data.

FIG. 14 illustrates an exemplary organization of media data supportinginformation.

FIG. 15 illustrates exemplary media data characterizations, including alook-ahead region, that indicate differing media data manipulationphases.

FIG. 16 is an exemplary sequence diagram that involves a look-aheadprocedure between a scheduler and a sender.

DETAILED DESCRIPTION

FIG. 2 is an exemplary environment 200 for media data disseminationarchitecture 202. Environment 200 includes, in addition to media datadissemination architecture 202, a network 204 and multiple clients206(1), 206(2) . . . 206(m). Clients 206 are capable of receiving andprocessing media data (not separately shown in FIG. 2). Clients 206 mayoptionally be capable of displaying and/or causing to be displayed themedia data. Such media data usually includes both audio and visualinformation, as well as optionally control information to facilitate theprocessing and/or display thereof. Although only three clients 206(1, 2,m) are explicitly depicted in FIG. 2, clients 206 may actually number inthe tens to hundreds of thousands (e.g., “m” can equal 10,000 to 100,000or more).

As illustrated, media data dissemination architecture 202 is coupled tonetwork 204 by one or more connection links 208. Clients 206 are coupledto network 204 by one or more network links 210. Consequently, mediadata dissemination architecture 202 is in communication with and coupledto clients 206 via network 204. Network 204 is capable of supportingunicast communications and optionally multicast communications incertain implementations described further herein below.

In operation of a described implementation, a particular client 206requests that a desired media data asset be sent to it. Media datadissemination architecture 202 receives the request and sends therequested media data asset to the particular client 206 using repeatedunicast communications over connection links 208, network 204, andnetwork links 210. The architecture of media data disseminationarchitecture 202 is adapted to effectuate a distributed sending of mediadata to clients 206 so as to reduce communication traffic bandwidthand/or bottlenecks that are otherwise internal to media datadissemination architecture 202.

Architecture for Distributed Sending of Media Data

FIG. 3 illustrates exemplary media data dissemination architecture 202that includes devices 302 for distributed sending of media data. Mediadata dissemination architecture 202 includes multiple devices 302(1),302(2) . . . 302(n). Each respective device 302(1, 2 . . . n) includes arespective sender 304(1, 2 . . . n) and a respective scheduler 306(1, 2. . . n). Media data dissemination architecture 202 also includes aswitch 308 and a mass storage of media data (MD) 310.

As illustrated, media data dissemination architecture 202 is coupled tonetwork 204 by connection links 208 at switch 308. Switch 308 is coupledto each respective device 302(1, 2 . . . n) by respective links 314(1, 2. . . n) that are internal to media data dissemination architecture 202.Switch 308 is also coupled to mass storage of media data 310.

Mass storage of media data 310 stores a sizable quantity of media datain some type of mass storage device such as a disk array. The disk arraymay be, for example, a redundant array of independent disks (RAID), afibre channel storage system, and so forth. By way of example only, massstorage of media data 310 may have a storage capacity of 5-6 terabytes(TBs). Nevertheless, a significant percentage of the total quantity ofmedia data is cached at devices 302 so that mass storage of media data310 is primarily accessed to provide media data for unexpected and/orrare media data asset requests and to rebuild failed devices 302.

In a described implementation, media data is cached in random accessmemory (RAM) (not specifically shown in FIG. 3) of devices 302(1, 2 . .. n). By way of example only, if there are 500 devices 302 (e.g., ifn=500) and each machine includes 4 gigabytes (GBs) of RAM, approximately2 TBs of RAM is available for RAM caching of the media data. Each device302(1, 2 . . . n) caches a proportion of the total percentage of mediadata that is cached by devices 302.

A hashing approach is used to allocate different media data blocks todifferent devices 302. This hashing approach and associated media dataorganization are described further below in the section entitled “MediaOrganization for Distributed Sending of Media Data”. This media dataorganization addresses the differences and relationships between andamong media data segments, media data assets, media data blocks, andmedia data sub-blocks.

Each respective sender 304 is associated with the media data that iscached by its respective device 302. Each sender 304 is responsible forstoring the media data that is allocated thereto and for sending themedia data in a distributed fashion to clients 206 responsive torequests from any scheduler 306 of schedulers 306(1, 2 . . . n).

Each scheduler 306 is associated with multiple clients 206. For eachgiven client 206 with which a particular scheduler 306 is associated,that particular scheduler 306 is responsible for meeting the media datadissemination requests of the given client 206. To that end, theparticular scheduler 306 may have knowledge of potential choke pointswithin network 204 and along network links 210 that can affect theability to disseminate media data to the given client 206 in a timelymanner. Hence, the particular scheduler 306 is capable of consideringthese choke points when scheduling the sending of media data fromsenders 304 to its associated clients 206. A choke-point related U.S.patent application entitled “Media Stream Scheduling for Hiccup-FreeFast-Channel-Change in the Presence of Network Chokepoints” with aninventorship of Dustin L. Green was filed on Oct. 10, 2003, and assignedU.S. patent application Ser. No. 10/683,132. This U.S. patentapplication Ser. No. 10/683,132, which has a common assignee ofMicrosoft Corporation with that of the instant Patent Application, ishereby incorporated by reference in its entirety herein.

Because the media data is striped across devices 302, a requestingscheduler 306 makes media data sending requests to multiple senders 304.When it is time to schedule delivery of a portion of a media data blockthat is associated with a particular sender 304, a requesting scheduler306 transmits a send request to that particular sender 304 using links314 and switch 308. The send request designates a destination client 206and stipulates/schedules a desired portion of a media data block(including a sub-block(s), sub-block range, or portion(s) thereof).

The particular sender 304 then forwards the stipulated media datasub-block to the designated client 206 using a link 314 and switch 308without routing the media data sub-block through the requestingscheduler 306 and without sending the media data sub-block throughswitch 308 twice. An exemplary set of messages for requesting andsending media data blocks is described further below, including withreference to FIG. 5 in this section.

As illustrated, a sender 304 and a scheduler 306 are included at eachdevice 302. However, because senders 304 and schedulers 306 arelogically separate, they may also be physically separate as well. Forexample, a number of devices 302 or each device 302 may alternativelyinclude a sender 304 or a scheduler 306 (but not both). Nevertheless, adescribed implementation includes a sender 304 and a scheduler 306 oneach device 302 so as to use relatively efficiently the processing,memory, and link bandwidth resources of each device 302.

Also illustrated for each respective sender 304(1, 2 . . . n) andscheduler 306(1, 2 . . . n) is a respective interface 312 (1, 2 . . . n)that is shared by both. Alternatively, each may have its own interface312 and link 314 to switch 308. By way of example only, switch 308 maybe realized using a Gb Ethernet switch with hundreds of ports. Each suchport corresponds to a link 314 and provides a two-way communicationbandwidth of 1 Gb/s per direction that is shared by senders 304 andschedulers 306 for both protocol and media data-related communications.

Alternatively, schedulers 306 (and therefore senders 304) may employ aseparate internal communication network with a separate switch (e.g., a100 Mb Ethernet switch) or router that provides a separate communicationavenue for protocol messages that are being sent between and amongschedulers 306 and senders 304. Exemplary separate linkages 314 areillustrated in FIG. 4 for schedulers 306 and senders 304 and maycorrespond to separate interfaces 312 and/or to separate communicationnetworks that are internal to media data dissemination architecture 202.

FIG. 4 illustrates exemplary devices 302 for distributed sending ofmedia data 406 that is striped across devices 302. Device 302(1)includes one or more processors 402(1) and at least one memory 404(1).Although not separately indicated as such, memory 404(1) includesprocessor-executable instructions that may be executed by processor402(1) to perform function(s) as described generally further below.Memory 404(1) may be realized as a combination of memories of differingtypes, features, and functions. Generally, however, memory 404 is arelatively low-latency memory with relatively greater throughput ascompared to mass storage of media data 310, which is a relativelyhigh-latency memory with relatively less throughput.

In a described implementation, these processor-executable instructionsmay comprise hardware, firmware, software, some combination thereof, andso forth. Modules having processor-executable instructions that arestored as parts of memory 404(1) include: sender 304(1), scheduler306(1), and media data (MD) 406(1). Media data 406(1) comprises a datastructure type module of processor-executable instructions. In adescribed implementation, media data 406(1) is stored in RAM. Each ofthese modules is described further herein below.

Sender 304(1) is coupled to switch 308 using link 314(1)-MD for mediadata communication, and scheduler 306(1) is coupled to switch 308 usinglink 314(1)-P for protocol communication. Scheduler 306(1) thustransmits and receives protocol-related messages via link 314(1)-P.Sender 304(1) sends media data 406(1) from device 302(1) via link314(1)-MD. Links 314(1)-P, as represented by optional links 314(1)-P*,may be realized as a separate network with a separate switch that isdedicated to protocol communications between senders 304 and schedulers306. If no other separate link 314(1) exists (e.g., link 314(1)-P*) forsender 304(1), sender 304(1) may also transmit and receiveprotocol-related messages via link 314(1)-MD.

Media data 406(1) includes a media data block for media data 408× and amedia data block for media data 408Y. Specifically, media data 406(1)includes media data block 408(X) and media data block 408(Y). Sender304(1) is therefore adapted to store media data blocks 408(X) and 408(Y)and is capable of sending them to clients 206 responsive to requeststhat are received from schedulers 306.

Device 302(2) includes analogous components to those of device 302(1).These components include one or more processors 402(2) and at least onememory 404(2), which includes sender 304(2), scheduler 306(2), and mediadata 406(2). However, media data 406(2) includes different media datablocks for media data 408X and 408Y as compared to those media datablocks (i.e., media data blocks 408(X) and 408(Y)) that are stored aspart of media data 406(1) at device 302(1).

As illustrated, sender 304(2) stores as part of media data 406(2) mediadata block 408(X+1) and media data block 408(Y+1). Media data block408(X+1) follows media data block 408(X) along a stream of media data408×. Media data block 408(Y+1) follows media data block 408(Y) along astream of media data 408Y. Thus, media data is striped (but notnecessarily linearly) across RAM as media data 408X and media data 408Yare striped across RAM portions of memories 404(1) and 404(2) of devices302(1) and 302(2), respectively. In a described implementation, mediadata is striped at a media data block granularity. Distributed sendingof media data 408X is described further below with reference to FIG. 5.

The illustration of FIG. 4 is not intended to convey any necessarilysequential correspondence between sequential media data blocks anddevices 302. In other words, devices 302(1) and 302(2) storing mediadata blocks X/Y and X+1/Y+1, respectively, is an example only.Sequential media data blocks are not necessarily sequentially striped,and they are in fact unlikely to be sequential using a “hash striping”approach as described herein below with particular reference to FIG. 11in the section entitled “Media Organization for Distributed Sending ofMedia Data”.

FIG. 5 illustrates an exemplary approach 500 to distributed media datadissemination that includes messages 504 and 506 used by multipledevices 302. The distributed sending of media data asset 408X isperformed on behalf of a client 206 that is associated with scheduler306(n) and that has requested delivery of media data asset 408X. A key502 indicates that open-tipped arrows represent protocol messages 504and that solid-tipped arrows represent media data messages 506.

In a described implementation, scheduler 306(n) transmits a send requestprotocol message 504(1) to sender 304(1) via switch 308. Send requestprotocol message 504(1) stipulates media data block 408(X) anddesignates the client 206 (e.g., using an identifying network address)to which the stipulated media data block 408(X) is to be sent. Inresponse, sender 304(1) sends media data message 506(1) to or toward thedesignated client 206 via switch 308 without directing it through sender304(n) or device 302(n). Media data (content) message 506(1) includesthe stipulated media data block 408(X).

Similarly, scheduler 306(n) transmits a send request protocol message504(2) to sender 304(2) via switch 308. Send request protocol message504(2) stipulates media data block 408(X+1) and designates the client206 to which the stipulated media data block 408(X+1) is to be sent. Inresponse, sender 304(2) sends media data message 506(2) to or toward thedesignated client 206 via switch 308 without directing it through sender304(n) or device 302(n). Media data (content) message 506(2) includesthe stipulated media data block 408(X+1).

Messages 504 and 506 of FIG. 5 omit some detail for the sake of clarity.For example, as noted above, messages 504 and 506 may actually bedirected to a portion or sub-block of an entire media data block 408(X)or 408(X+1). Moreover, each individual message 504 may request thesending of multiple such portions or sub-blocks of an entire media datablock. Media data sub-blocks are described further below with particularreference to FIG. 10.

Employing distributed media data dissemination approach 500 amelioratespotential problems with bottlenecks. Unfortunately, bottlenecks canstill occur, especially with media data blocks that are currently verypopular. By distributing (e.g., using a hash striping) the media data atthe media data block level across multiple devices 302, distributedsending can be employed which concomitantly distributes workload andbandwidth constraints across many devices 302 and links 314,respectively. Unfortunately, this distribution can be jeopardized forvery popular media data blocks.

For example, live media data presentations can result in a significantportion of clients 206 requesting not only the same media data assetsimultaneously but the actual same media data block simultaneously aswell. Because the entire media data block is on a single device 302 withlimited sending bandwidth, it can be difficult to service all interestedclients 206 simultaneously. Consequently, for popular media data blocksgenerally and for live media data presentations specifically,multicasting and/or block replication is utilized in a describedimplementation.

Multicasting can alleviate bottlenecks from simultaneous requests for asingle media data block by enabling a device 302 to send one copy (or areduced number of copies) of each sub-block of a media data block. Themedia data sub-block is sent to a single replication point (or a lowerednumber of replication points compared to the total number of interestedclients 206) that is downstream from switch 308 and within amulticast-capable implementation of network 204.

An intention to utilize multicasting may be determined ahead of timemanually, especially for live media data presentations such as knownsporting events. Media data dissemination architecture 202 may also oralternatively be capable of monitoring an instantaneous popularity ofmedia data blocks (or entire assets) and migrating from aunicast-media-data-sending to a multicast-media-data-sending when theinstantaneous popularity reaches a predetermined level (e.g., 40%, 60%,etc.) for a single media data block.

Generally, media data block replication can alleviate bottlenecksarising from simultaneous requests for a single media data block byenabling multiple devices 302 to send the replicated block. For livemedia data presentations, the multiple devices 302 may acquire thereplicated blocks via multicasting. The multiple devices 302 that have acopy of the replicated media data block may be all or only a sub-set ofthe total devices 302(1 . . . n). Media data block replication isdescribed further below with particular reference to FIGS. 11-13.

Although bottlenecks are usually not an issue for non-live media datapresentations, bandwidth constraints that are internal to media datadissemination architecture 202 are still addressed to facilitate therelatively reliable and efficient distributed sending of media datapresentations generally. For example, each link 314 has a maximumoutgoing bandwidth, and each respective sender 304 is receiving multiplesend request protocol messages 504 from different schedulers 306.Consequently, no single scheduler 306 is aware of how much bandwidth agiven sender 304 has available. This per-sender 304/per-link 314bandwidth issue is addressed below in the section entitled “PriorityMechanism for Distributed Sending of Media Data”.

Priority Mechanism for Distributed Sending of Media Data

In a described implementation, a priority mechanism enables relativelyefficient and full utilization of available media data sending bandwidthfrom senders 304.

FIG. 6 illustrates a first aspect 600 of an exemplary approach to apriority mechanism for distributed sending of media data. A scheduler306 and a sender 304 are shown coupled to a link 314. Link 314 has afinite outgoing bandwidth 602. Sender 304 receives multiple sendrequests 604(1), 604(2), 604(3) . . . 604(v).

Each send request 604 can correspond to a send request protocol message504 that stipulates a media data block (including a portion thereof) anddesignates a client 206 to which at least a portion of the media datablock is to be sent. Each send request 604 originates at and istransmitted from a scheduler 306 of multiple schedulers 306 (that arenot shown in FIG. 6).

In a described implementation, sender 304 includes a send bandwidth 606,a send request ranker 608, and a threshold priority ascertainer 610.Send bandwidth 606 is the amount (e.g., the percentage) of outgoingbandwidth 602 that is partitioned for use by sender 304 when sendingmedia data to clients 206. Typically, send bandwidth 606 is less thanoutgoing bandwidth 602 in order to reserve some of outgoing bandwidth602 for protocol messages 504 and to over-provision the system to reducethe likelihood of a streaming failure. For example, send bandwidth 606may be 95% of outgoing bandwidth 602.

Send request ranker 608 is adapted to rank send requests 604 in order ofpriority. Each send request 604 is associated with a unique globalpriority. The unique global priority is calculated from one or more of avariety of factors. First, priority is dependent on whether therequested media data sub-block is deadline data or early data. Deadlinedata is media data that must be sent during a current timeslot or astreaming failure is extremely likely to occur. Early data is data thatis being sent prior to what would otherwise be a deadline timeslot.Deadline data has a higher priority than early data.

Priority is also dependent on whether two (or more) devices 302 storethe requested media data sub-block. For example, for highly popularmedia data blocks, the media data is replicated onto a second device302. Media data that is replicated has a lower initial priority thanmedia data that is on a single device 302 or, more generally, that isassociated with only a single sender 304. However, if neither device 302of two devices 302 would send the replicated media data if two possibledevices were factored into or considered in the prioritization, oneparticular device 302 is picked and that particular device 302 isinformed that there is only one capable device.

The part of the priority determination process that reflects the numberof devices 302 (or senders 304) that are capable of sending the data istermed the “option count”. The option count equals the number of devices302 that store a copy of a given media data block. Generally, media datasends that can originate from relatively fewer devices (corresponding toa relatively lower option count) are higher priority than media datasends that can originate from relatively more devices (corresponding toa relatively higher option count). Replication and popularity aredescribed further below in the section entitled “Media Organization forDistributed Sending of Media Data”.

The priority determination process also involves consideration of howearly a data request is when it is not a deadline data request. Forexample, relatively sooner early data is assigned a higher priority thanrelatively later early data. By way of explanation, sooner early data isused by a client before later early data. Thus, although both are aftera deadline, sooner early data is closer in time to the deadline thanlater early data. Deadline data and early data are discussed furtherbelow in the section entitled “Scheduling Scheme for Distributed Sendingof Media Data”.

Priority is also dependent, at least in cases that would otherwiseresult in ties, on network addresses and/or other unique ID values. Forexample, the address of device 302, the address of the scheduler 306,the address of the sender 304 (e.g., if separate from device 302 and/orscheduler 306), the address of the destination client 206, somecombination thereof, etc. may be factored into the priority calculation.For instance, a media data send on behalf of a device 302 (or ascheduler 306 thereof) with a lower internet protocol (IP) address maybe assigned a higher priority than a media data send on behalf of adevice 302 with a higher IP address.

Other prioritization factors may also be considered when calculating theunique priority of a send request 604. In short, an exemplary list ofprioritization factors is:

-   -   Deadline data versus early data, with deadline data having a        higher priority;    -   Fewer capable senders versus many capable senders, with fewer        capable senders having a higher priority;    -   Sooner early data versus later early data (which may be        established as a continuum), with sooner early data having a        higher priority;    -   Scheduler ID (which can be formed from, e.g., the IP address        plus a port number or some other ID that is unique to each        individual scheduler 306);    -   Uniqueifying request ID (which is assigned by a scheduler 306 to        keep all of its own requests unique from one other—and therefore        guaranteed unique from other scheduler's requests because of the        different scheduler IDs); and    -   so forth.

In a described implementation, each scheduler 306 calculates theassociated unique priority of each send request 604 that it istransmitting, and this associated unique priority is included in eachsend request 604. The transmitting scheduler 306 is aware of whether therequested media data is deadline data, early data, how soon the earlydata is, and so forth. However, senders 304 may alternatively performthe priority calculation, especially if any relevant factors that theydo not otherwise have are provided to senders 304.

Send request ranker 608 therefore uses respective associated uniquepriorities of respective send requests 604 to rank the associated sendrequests 604. Send requests 604 are ranked from most important to leastimportant. As used herein, highest priority equates to most important,greater priority equates to more important, and lower priority equatesto less important.

Threshold priority ascertainer 610 is configured to ascertain athreshold priority for sender 304 based on the ranked send requests 604and responsive to send bandwidth 606. Specifically, bandwidth that isconsumed by send requests 604 is accumulated first from those sendrequests 604 that are more important and then second from those sendrequests 604 that are less important. When the accumulated bandwidthmeets or exceeds send bandwidth 606, a send request cutoff with respectto send requests 604 is detected. The associated unique priority of the“last” (i.e., least important) send request 604 to be within sendbandwidth 606 and thus to make the send request cutoff is ascertained tobe the threshold priority. The send request cutoff and determination ofthe threshold priority are described below with particular reference toFIG. 7.

FIG. 7 illustrates a second aspect 700 of the exemplary approach to apriority mechanism for distributed sending of media data. Send bandwidth606 is depicted as a tube that can be filled to the brim, but any sendrequests 604 that spill over are not currently sendable. Four sendrequests 604(1, 2, 3 . . . v) are shown after ranking by send requestranker 608 from most important (at the bottom) to least important (atthe top). Although only four send requests 604 are explicitly picturedin priority ranking 706, many such send requests 604 are usually handledby sender 304 during each timeslot.

As illustrated, send request 604(1) is associated with (unique) priorityA, and send request 604(2) is associated with priority B. Also, sendrequest 604(3) is associated with priority C, and send request 604(v) isassociated with priority D. In this illustrated example, priority A isless important than priority B, priority B is less important thanpriority C, and priority C is less important than priority D. Hence,send request 604(v) is applied to send bandwidth 606 first.

Consequently, bandwidth accumulation starts with send request 604(v) andproceeds upwards to send request 604(3) and beyond until a send request604(2) is reached that cannot fit entirely within the send bandwidth606. At this point, send requests 604 that do not fully fit within sendbandwidth 606 (i.e., send requests 604(1) and 604(2) in the illustratedexample) may be discarded by the sender 304 that is implementing thisaspect of the priority mechanism because their send requests cannot behonored during the current timeslot.

However, send request 604(2) can alternatively be only partiallydiscarded by the sender such that part of the requested media data canbe sent during the current timeslot. Partially discarding a send requestbut honoring the remainder is likely a minor optimization in mostscenarios, so the probable increased throughput is weighed againsthandling the increased complexity of dividing send requests.

It should be noted that if each media data sub-block is of an identicalsize and if send requests all specify a single data sub-block (which isnot the case for a described implementation and usually not the optimalimplementation for maximizing real-world throughput), then bandwidthaccumulation may be accomplished by dividing send bandwidth 606 by thebandwidth consumption size of each media data sub-block to produce aselected number of send requests 604 that are to be honored. Relativelymore important send requests 604 equal in number to the selected numberare then identified as being below a send request cutoff 702.

Send request cutoff 702 corresponds to the least important send request604 of received send requests 604(1, 2, 3 . . . v) that can be includedin the current round of send media data messages 506 without exceedingsend bandwidth 606. Any send request 604 that fully or partially exceedssend bandwidth 606 is above send request cutoff 702 and cannot(according to operational protocols) be honored (at least completely)for sending in the current round of send media data messages 506.

Threshold priority 704 for sender 304 for the current timeslot or roundunder consideration corresponds to the unique priority of the lowestpriority send request 604 that fits within the predetermined sendbandwidth 606. In an alternative implementation, threshold priority 704can correspond to the unique priority of the highest priority sendrequest 604 that does not fit within the predetermined send bandwidth606, as long as schedulers 306 know which scheme senders 304 are usingto communicate send request cutoff 702. Also, if all of the sendrequests 604 that are received at a given sender 304 do not equal orexceed the given sender's send bandwidth 606, that given sender 304 mayconsider (and therefore broadcast) a threshold priority 704 that equalsa minimum possible threshold priority value.

In the illustrated example, threshold priority 704 corresponds topriority C of send request 604(3). This threshold priority 704 istransmitted by sender 304 to multiple schedulers 306 as an indicator ofwhich send requests 604 are not being honored/selected for sending bythat sender 304 in the current timeslot and which are beinghonored/selected, at least up to the point at which threshold priority704 (e.g., an initial threshold priority 704) is being determined.

Schedulers 306 do not know which send requests 604 will actually behonored/selected for sending by a given sender 304 in the currenttimeslot until they receive a final threshold priority 704 from thatgiven sender 304. Initial, intermediate, and final threshold priorities704 are described further below with particular reference to FIG. 9.From receipt of a final threshold priority 704, schedulers 306 know (i)that any send request 604 at least as important as the final thresholdpriority 704 for which they have received an ACK notification washonored/selected, (ii) that any send request 604 at least as importantas the final threshold priority 704 for which they have received neitherACK notification nor non-sent notification may or may not have beenhonored/selected, (iii) that any send request 604 at least as importantas the final threshold priority 704 for which they have received anon-sent notification was not honored/selected for some reason (e.g.,due to late reception by sender 304 of send request 604 (i.e., sendrequest 604 may be received after the cutoff time at which the finalthreshold priority 704 is determined and just before media blocksub-block(s) are sent)), and (iv) that any send request 604 lessimportant than the final threshold priority 704 (as described above) wasnot honored/selected regardless of whether it had previously received anACK notification for that send request.

Threshold priority 704 can be transmitted by sender 304 as a unicastcommunication or as a multicast communication. For example, sender 304can unicast threshold priority 704 to each scheduler 306 that submitteda send request 604, to each scheduler 306 whose submitted send request604 is not being honored, to each scheduler 306 whose submitted sendrequest 604 is being honored, to schedulers 306 generally (e.g.,“broadcast” with multiple unicast communications), and so forth.Alternatively, sender 304 can multicast threshold priority 704 toschedulers 306 generally (e.g., broadcast with a multicastcommunication). An implementation of how a scheduler 306 may utilize thetransmitted threshold priority 704 is described below with reference toFIG. 8 (and further below with reference to FIG. 9).

FIG. 8 is a flow diagram 800 that illustrates an exemplary method for apriority mechanism for distributed sending of media data. Flow diagram800 includes eleven (11) blocks 802-822. Although the actions of flowdiagram 800 may be performed in other environments and with a variety ofhardware and software implementations, FIGS. 5-7 are used in particularto illustrate certain aspects and examples of the method. For example, ascheduler 306 (of a first device 302) may perform the actions of blocks802-810 and 822, and a sender 304 (of the first or a second device 302)may perform the actions of blocks 812-820.

At block 802, a send request having a unique priority is transmitted.For example, scheduler 306 may transmit a send request 604(*) that isassociated with a priority that differs from every other send request604 (at least in a current timeslot).

At block 812, the send request having the unique priority is received.For example, sender 304 may receive send request 604(*) from scheduler306. At block 820, an ACK of receipt of the send request is transmittedto the scheduler. For example, sender 304 may transmit an ACK thatacknowledges receipt of send request 604(*) to scheduler 306, optionallyin dependence upon the unique priority of send request 604(*) and acurrent threshold priority 704 of sender 304. At block 804, the ACK ofreceipt of the send request is received from the sender. For example,scheduler 306 may receive the ACK that acknowledges receipt of sendrequest 604(*) from sender 304. Receipt of an ACK for send request604(*) may also be a factor in the sending selection determination ofblock 808, which is described below.

Sender 304 may also receive multiple other send requests 604(1, 2, 3 . .. v) from scheduler 306 and other schedulers. At block 814, multiplesend requests are ranked according to respective unique priorities. Forexample, a send request ranker 608 of sender 304 may rank multiple sendrequests 604(1, 2, 3 . . . v), including send request 604(*), into apriority ranking 706.

At block 816, a threshold priority is ascertained based on the rankedsend requests and responsive to a send bandwidth. For example, athreshold priority ascertainer 610 may ascertain a threshold priority704 based on the ranked send requests 604(1, 2, 3 . . . v) in priorityranking 706 and responsive to a send bandwidth 606. For instance, apriority of a send request 604, such as priority C of send request604(3), may be ascertained to be threshold priority 704 if the sendrequest 604 is associated with a lowest priority from among those sendrequests 604 that fall below (or more generally within) a send requestcutoff 702 as set or otherwise established by send bandwidth 606.

At block 818, the ascertained threshold priority is broadcast. Forexample, sender 304 may multicast threshold priority 704 to schedulers306. The ascertained and broadcast threshold priority is received bymultiple schedulers. For example, scheduler 306 may receive thresholdpriority 704. It should be noted that sender 304 continues its thresholdpriority analysis based on newly-arriving send requests 604 as indicatedby arrow 824 and as is described further below with reference to FIG. 9.

At block 806, the unique priority of the send request is compared to thethreshold priority. For example, the priority associated with sendrequest 604(*) may be compared to threshold priority 704. At block 808,it is determined if the send request has been selected for sending(i.e., will be honored) based on the comparing (e.g., assuming that anACK notification has been received for the send request (as in block804) and that the received threshold priority is the final thresholdpriority of the timeslot). For example, scheduler 306 may determine thatsend request 604(*) has been selected for sending if the priorityassociated with send request 604(*) is greater than threshold priority704.

In a described implementation, an ACK notification is transmitted from asender 304 to a scheduler 306 (as in block 820) when the thresholdpriority 704 of the sender 304 does not preclude sending the media dataof a send request 604. If the current threshold priority 704 alreadydoes preclude the media data sending, then no ACK is sent because thebroadcast threshold priority 704 will inform scheduler 306 that it isirrelevant whether or not its send request 604 was received by thesender 304 because the requested media data is not being sentregardless.

Although possible, entirely eliminating these ACK notifications for sendrequests (as illustrated by blocks 820 an 804) is probably inferior inmost scenarios in which reliable message delivery cannot be assumed(e.g., in network environments with no intrinsic reliability alreadybuilt in). More specifically, such elimination is inferior when thepacket loss rate is significant (e.g., above approximately half apercent) because requestors would not have any way to know whichrequests should be retransmitted.

Thus, more generally, scheduler 306 may determine that send request604(*) has been selected for sending if the following confirmingconditions are met: (i) the priority associated with send request 604(*)is greater than or equal to threshold priority 704, (ii) an ACKnotification has arrived for send request 604(*) from sender 304, and(iii) threshold priority 704 is marked as final. Scheduler 306 maydetermine that send request 604(*) has not been selected for sending ifthe priority associated with send request 604(*) is lower than thresholdpriority 704.

Further, scheduler 306 may determine that it is not yet possible todetermine whether send request 604(*) was or was not selected forsending if (i) the priority associated with send request 604(*) isgreater than or equal to threshold priority 704, (ii) no ACKnotification has been received for send request 604(*) from sender 304,(iii) no non-sent notification has been received for send request604(*), and (iv) threshold priority 704 is marked as final. In such acase, which is rare for most scenarios of interest, scheduler 306 mayassume that the send request was or was not selected for sending,whichever is more appropriate given the costs involved if an incorrectassumption is made. Also, in some implementations, scheduler 306 maycontinue to re-send send request 604(*) even after a final thresholdpriority 704 is received simply in order to determine whether sendrequest 604(*) was selected for sending; however, this is beneficialprimarily when the cost of an incorrect assumption is enough to justifythe extra memory and processing requirements necessary to keep track ofmore than one timeslot at a time.

Thus, in short, if all of the above confirming conditions (i)-(iii) arenot met, scheduler 306 delays determination of whether send request604(*) has been selected for sending. In the unlikely event that atimeout occurs before it can be determined whether send request 604(*)has been selected for sending, scheduler 306 may, possibly incorrectly,(i) assume that send request 604(*) was not selected, in which casescheduler 306 will re-request it later (with the possible result of itbeing sent twice), or (ii) assume that send request 604(*) was selected,in which case the client will send a retry request to fill in anymissing media data.

Continuing with flowchart 800, if it is determined that “Yes” the sendrequest has been selected for sending (at block 808), then at block 822an acknowledgment (ACK) from a destination client is awaited orcommunications from the destination client are monitored for a potentialNACK. For example, scheduler 306 may await a reception acknowledgmentmessage from a client 206 that is the designated destination for mediadata stipulated in send request 604(*), where the media data is beingsent to client 206 by sender 304. Alternatively, scheduler 306 maymonitor communications from a client 206 that is the designateddestination for media data stipulated in send request 604(*), where themedia data is being sent to client 206 by sender 304, for potentialNACKs sent by client 206 when it does not receive something it expects.

As indicated by the two options in the dashed block 822, some deliveryprotocols do not require an ACK from clients 206, and they may insteadrely on NACK messages to fill in gaps. In such cases, scheduler 306takes no special affirmative action after the “Yes” branch from block808 beyond noting that send request 604(*) was selected for sending andthat the referenced media data therefore does not need to be sent againunless a client 206 specifically requests some or all of the referencedmedia data. Hence, with a NACK-based protocol, scheduler 306 may engagein relatively passive monitoring for NACK communications from clients206 once a send request 604 has been selected for sending.

If, on the other hand, it is determined that “No” the send request hasnot been selected for sending (at block 808), then at block 810 it ischecked if another sender has a threshold priority that is less than theunique priority of the send request and/or less than the highestassignable unique priority for the send request. For example, scheduler306 may check another sender 304′ that also stores the stipulated mediadata to see if the threshold priority 704′ of sender 304′ is lower thanthe priority already associated with send request 604(*) and/or is lowerthan the highest priority that can be assigned to send request 604(*).If there is an alternative sender 304, another send request 604(*)′ withthe same send priority or an increased send priority (e.g., iflegitimately adjustable and necessary to find a usable sender) istransmitted from scheduler 306 (i.e., the method of flow diagram 800 maycontinue at block 802 with a new send request and a new target sender ifthe send priority is not increased and either the same or a new targetsender if the send priority is increased).

One way to legitimately adjust the send priority of a send request 604is to adjust the option count (i.e., the number of devices 302 thatcurrently cache the desired media data) downward to a value lower thanthe initially or previously-used value. In other words, the number ofsenders that are potentially capable of sending the referenced mediadata block may be “virtually” decreased for purposes of increasing asend priority. This can be allowed because otherwise a situation mightarise in which there are two or more senders that could send data for ablock were the send request to have a sufficiently high priority, butnone of them accept the send request if the option count is set to theactual number of senders that are theoretically capable of sending mediadata of the media data block.

In an alternative described implementation with respect to block 810,the unique priority may be adjusted (e.g., by lowering an option count)after block 808 but prior to block 810. If the increased unique priorityof the would-be send request results in a capable sender having a lowerthreshold priority, then a new send request may be sent to this othercapable sender (e.g, by repeating the action(s) of block 802).

In short, one or more other senders 304 may store a replicated copy ofthe media data as noted above and as described further below in thesection entitled “Media Organization for Distributed Sending of MediaData”. The threshold priority 704 of each sender 304 having such areplicated copy may be compared to the current priority and/or thehighest priority that can be legitimately associated with send request604(*) (e.g., by temporarily using a decreased option count, such as anoption count of one). If a threshold priority 704′ that is lower isdiscovered, then a second send request 604(*)′ may be transmitted to thesender 304′ thereof, with the option count set to a low enough valuethat the send request 604(*) has an associated priority that is higher,or more important, than threshold priority 704′. This sendingnegotiation, which is described below with reference to FIG. 9, mayproceed until send bandwidths 606 of multiple senders 304 are filledand/or until a time available for negotiating a current timeslotexpires.

FIG. 9 illustrates an exemplary send negotiation per-time-slot 900 thatinvolves a priority mechanism for distributed sending of media data. Foreach sending timeslot, a negotiation is performed in an attempt to fillup the send bandwidth 606 of each sender 304 at each device 302 withoutover-filling the available bandwidth with deadline data send requestsalone. The architecture endeavors to utilize a significant percentage(if not all) of each send bandwidth 606 through a number of schemes.

These bandwidth utilization schemes include, for example, media datareplication that increases the potential number of capable senders 304for some or all media data blocks. (Media data replication alsoincreases the total number of clients 206 that can be serviced.) Anotherscheme relates to sending media data early, before the sending deadlineof the media data, so that media data sub-blocks that must (if astreaming failure is to be avoided) be sent to clients 206 at or by afuture deadline timeslot are sent in current timeslots.

Yet another bandwidth utilization scheme entails schedulers 306, whentaken in the aggregate, being prevented from allocating more than acertain percentage (e.g., 85%) of the overall available send bandwidthcapacity when totaling individual deadline-send bandwidths 606 acrossthe multiple senders 304. Each scheduler 306 is configured with a“deadline-only chokepoint” that affects deadline send allocations infuture timeslots, but it is not considered when allocating early sendsin the current timeslot. While this does not guarantee that every oreven any individual sender 304 will be requested to send no more than85% of its individual send bandwidth 606 (counting only deadline sendsthat cannot be sent from any other device 302), it does greatly reducethe probability that any sender 304 will be requested to send more than100% of its send bandwidth 606 (counting only deadline sends that cannotbe sent from any other device 302).

Because senders 304 are often requested to utilize the remainder oftheir send-bandwidth 606 by sending early data and thus total sendscounting both deadline and non-deadline sends of a sender 304 oftenessentially fill send bandwidth 606 of that sender 304, it is apparentthat preventing the aggregate deadline data utilization of sendbandwidths 606 of multiple senders 304 from exceeding 85% (for example)does not preclude using the remaining 15% for early data. An exemplaryscheduling scheme and pipeline management approach is described furtherbelow in the section entitled “Scheduling Scheme for Distributed Sendingof Media Data”.

As illustrated, send negotiation 900 occupies at least part of atimeslot. Activities in the upper portion of send negotiation 900 areperformed by senders 304. Activities in the lower portion of sendnegotiation 900 are performed by schedulers 306.

Schedulers 306 initially determine what media data they wish to send towhich clients 206 based on deadlines, network choke points, and soforth. Based on these determinations, schedulers 306 formulate sendrequests 604 (not separately shown in FIG. 9). These send requests 604,which are based on default values instead of threshold priorityinformation, are transmitted as original send requests 902 fromschedulers 306. It should be noted that threshold priorities 704 for allsenders 304 at the beginning of each timeslot may be considered to be aminimum threshold priority value as set by a given system's design. Theintended individual senders 304 receive original send requests 902.

As described above with particular reference to FIGS. 6-8, respectivesenders 304 ascertain respective threshold priorities 704 responsive tothe original send requests 902. These ascertained respective thresholdpriorities 704 are transmitted from respective senders 304 as initialthreshold priorities 904. Schedulers 306 receive initial thresholdpriorities 904.

Responsive to the received initial threshold priorities 904, schedulers306 are able to evaluate options 906. In other words, empowered by theknowledge of initial threshold priorities 904 of senders 304 for thecurrent timeslot, schedulers 306 can evaluate options that facilitatesend bandwidth 606 utilization and/or media data dissemination toassociated clients 206. For example, if replicated media data forstipulated media data from a rejected send request 604 is present at adifferent sender 304 on a different device 302, a new send request 604that is intended for that different sender 304 may be formulated. Thisformulation is appropriate if the send priority of the rejected sendrequest 604 is already or can be adjusted to be (e.g., by lowering anoption count to produce a priority that is) higher than the thresholdpriority 704 of the different sender 304. Alternatively, media data witha later deadline that is not required by a destination client 206 untilan even later time may be stipulated in another send request 604,depending on threshold priorities of other senders 304 that areassociated with such later early data.

These new send requests 604 are transmitted as subsequent send requests908 from schedulers 306. The intended individual senders 304 receivesubsequent send requests 908. After adding those subsequent sendrequests 908 that fall within respective send request cutoffs 702 wherepossible, respective senders 304 ascertain new respective thresholdpriorities 704. These new ascertained respective threshold priorities704 are transmitted from respective senders 304 as intermediate(non-final) threshold priorities 910. Schedulers 306 receiveintermediate threshold priorities 910. In this manner for a describedimplementation, threshold priorities 704 for senders 304 increase withineach timeslot during send negation 900 but do not decrease. Thisfacilitates rapid stabilization and thus completion of send negotiation900.

As indicated by ellipses 912, send negotiation 900 may continue with theexchange of additional subsequent send requests 908 and correspondingintermediate* threshold priorities 910, until what would otherwise be anintermediate* threshold priority is marked as final. In other words, asthe negotiation time nears an end, senders 304 mark their thresholdpriorities as final threshold priorities 914 and transmit them toschedulers 306. After receiving final threshold priorities 914,schedulers 306 cease reevaluating sending options, formulating new sendrequests, and transmitting such new send requests. Schedulers 306 do,however, formulate and transmit new original send requests 902 in thenext succeeding timeslot. It should be noted that send negotiation 900also ceases (in a de facto cessation if not an official cessation) whenall schedulers 306 have determined that no media data underconsideration for the current timeslot has a maximum assignable prioritythat is greater than the individual schedulers' views of any usablesenders' current (but not necessarily final) threshold priority 704.

Although three subsequent send request 908 transmissions and threeintermediate threshold priority 910 transmissions are shown, none, one,two, or more than three may alternatively occur. If none occur, then theinitial threshold priority 904 is also the final threshold priority 914.It should be understood that there need be no formal rounds internal toeach timeslot. Send negotiation per time slot 900 is an organic processin which senders 304 and schedulers 306 communicate their new sendrequests 604 and current threshold priorities 704 until time expires anda current threshold priority 704 is marked final. Hence, the exchange ofmessages may be unsynchronized, may be overlapping, etc. between andamong different schedulers 306 and senders 304.

In a more-specific described implementation, the back and forthmessage-exchanging negotiation may happen several times, not just onceor twice. Senders 304 ACK send requests 604 that are greater than theircurrent threshold priority 704 while possibly updating their new currentthreshold priority 704 if a send request 604 is being received for thefirst time and if the threshold priority 704 is thereby affected.Schedulers 306 retransmit send requests 604 that still have a greaterunique priority than the scheduler's view of the sender's currentthreshold priority 704 if the send requests 604 have not previouslyresulted in a received ACK. NON_SENT messages may be sent when a sendrequest 604 (e.g., a subsequent send request 908) with a higher uniquepriority than a sender's final threshold priority 914 arrives at thesender 304 after transmission of its final threshold priority 914, butthis is optional. Schedulers 306 may send a NON_FINAL message to asender 304 after final threshold priority 914 transmissions if a “final”threshold priority from the sender 304 has not been received, but thistoo is optional. As a related option, senders 304 may resend their finalthreshold priority 914 if they receive a NON_FINAL message.

As described above, send negotiation 900 is eventually terminated sothat scheduled and accepted media data block portions 916 may be sentfrom senders 304 to clients 206. For example, send negotiation 900 maybe terminated after a predetermined period of time (e.g., some fractionof the timeslot) has elapsed. Any messages being passed with referenceto the current timeslot at the expiration of that period of time areignored, and original send requests 902 for the immediately succeedingtimeslot may be prepared and then transmitted. Other terminationprovisions may alternatively be instituted.

Media Organization for Distributed Sending of Media Data

FIG. 10 illustrates an exemplary logical organization 1000 of mediadata. A media data (MD) segment 1002 is divided into one or more mediadata assets 1004. By way of a content-oriented example, a media datasegment 1002 may correspond to a day of a news channel, nationallybroadcast programming for a major network for any given 24 hour periodor any other length period (e.g., 16 hours), a pay-per-view movie, andso forth. Each media data asset 1004 may be, for example, some dividedportion (e.g., a sub-region) of media data segment 1002 or the entiretyof media data segment 1002. Content-oriented examples of possible mediadata assets 1004 include a 30 minute portion of an all-news channel, anhour-long drama from network television, and so forth.

However, a media data asset 1004 is conceptually flexible depending onthe media data contents. For example, a media data asset 1004 may be arelatively short snippet of media data such as a movie short, anindividual news story, and so forth. On the other hand, a media dataasset 1004 may be a relatively long piece of media data such as a fullfeature-length movie. The full feature-length movie media data asset1004 may also happen to correspond to the entirety of its media datasegment 1002.

From a technical perspective, a media data segment 1002 may be acontinuous time line of media data from an encoder that can becontinuously decoded. If the encoder ceases encoding, significantlyalters its encoding parameters, or is changed to another encoder, adifferent media data segment 1002 is started. The entirety of eachdesired media data segment 1002 is typically retained. However, they maybe divided into sub-regions, which are capable of being referencedspecifically and individually, such as media data assets 1004 so thatpopularity can be focused to at least a per-media-data-asset 1004 level.Popularity may also be tracked at a finer level of detail, such as on aper-block basis, a per-half-hour basis, a per-sub-block basis, aper-byte basis, or at any other level of detail. Regardless of the levelof detail at which popularity is actually tracked, an estimatedpopularity of a specific block can be easily derived. Popularity isdescribed further below, especially with reference to FIGS. 12 and 13.

Each media data segment 1002 is sub-divided into one or more media datablocks . . . 1006(w−1), 1006(w), 1006(w+1) . . . Although most mediadata segments 1002 likely contain many media data blocks 1006, a givenmedia data segment 1002 may contain as few as three, two, or even asingle media data block 1006. In a described implementation, media datablocks 1006 are the level of media data granularity that is distributedacross devices 302. For example, the presence of distributed media datablocks 1006 is represented by media data blocks 408(X), 408(X+1),408(Y), and 408(Y+1) in FIGS. 4 and 5. Media data blocks 1006 areapproximately one megabyte (MB) in size, but other sizes mayalternatively be implemented. The media data block size for video datamay be different than the media data block size for audio data withinthe same segment. The media data block size may also be variable withina single stream, but this is rarely done.

Each media data block 1006, such as media data block 1006(w), is furthersub-divided into media data sub-blocks 1008(1), 1008(2), 1008(3) . . .1008(s). In a described implementation, media data sub-blocks 1008 arethe level of media data granularity that is specified for sending toclients 206 in a given timeslot. Hence, a media data sub-block 1008 isthe unit of media data that is stipulated for sending in each sendrequest 604 (e.g., is the level of media data granularity for sending).However, a single send request 504/604 can request the sending ofmultiple media data sub-blocks 1008, instead of only requesting thesending of a single media data sub-block 1008. In other words, althougheach media data sub-block 1008 may be packet-sized, a single sendrequest 604 can result in the sending of multiple such media datasub-blocks 1008 (and thus their packets) to a client 206.

FIG. 11 illustrates an exemplary approach 1100 to physically locatingmedia data that includes a hash table 1114. Media data block 1006(w) ofmedia data segment 1002 is shown being located on a device 302, which isdevice 302(2) in this example. Physically locating media data may referto locating media data (e.g., a media data block 1006) (i) for purposesof placing the media data in a device 302 location or (ii) for purposesof requesting that the device 302 having the media data location sendthe media data to a client 206.

In a described implementation, each media data segment 1002 andindividual media data block 1006 include or correspond to an identifyingnumber. As illustrated, media data segment 1002 includes a media datasegment number 1102, and media data block 1006(w) includes a media datablock number 1104. Each media data segment 1002 in the system may beassigned a unique media data segment number 1102 in a sequential fashionor by using globally unique IDs, or in some other fashion, as long asthey are unique. Within a given media data segment 1002, each media datablock 1006 is sequentially assigned a unique media data block number1104. The sequentially-assigned media data block numbers 1104 can alsobe used for video processing ordering.

Media data segment number 1102 and media data block number 1104 arecombined (e.g., added, concatenated, etc.) to derive a media dataidentification (ID) value 1106. Media data identification value 1106 isapplied to a hashing function 1108 to produce a media data hash value1110. In a described implementation, hashing function 1108 employs alinear feedback shift register (LFSR) with two 32-bit values (from mediadata segment number 1102 and media data block number 1104). However,alternative hashing functions 1108 may instead be employed, includingnon-LFSR functions and/or functions that use values of differentbit-lengths.

Media data hash value 1110 is mapped at mapping 1112 onto hash table1114. Hash table 1114 includes multiple bins 1116(1), 1116(2), 1116(3) .. . 1116(b). Specifically, media data hash value 1110 is mapped 1112onto a bin 1116 of hash table 1114. In a described implementation,mapping 1112 includes taking the remainder when media data hash value1110 is divided by the size “b” of hash table 1114. As illustrated,media data hash value 1110 is mapped 1112 to bin 1116(2) of hash table1114.

Each bin 1116 of hash table 1114 has an association 1118 with at leastone device 302. In an implementation that is described further belowwith reference to FIG. 13, each bin 114 has an association with twodevices 302. For example, a first device 302 may have a primary rolewith a given bin 1116 while a second device 302 has a secondary rolewith the given bin 116. The roles and associations with bins 1116 mayalternatively be defined with respect to senders 304, instead of actualdevices 302; an example having associations 1118 between bins 1116 andsenders 304 is described further below with reference to FIG. 13.

As illustrated, bin 1116(2) is associated 1118 with device 302(2).Hence, device 302(2) has a e.g. primary or secondary role with bin1116(2). Media data block 1006(w) is therefore locatable at device302(2) and stored in RAM thereof in a RAM striping implementation.

In a described implementation, the number “b” of bins 1116 is greaterthan the number “n” of devices 302. For example, if n=500, then b mayequal 131,071. As a result, and as indicated by the second arrowextending from hash table 1114 to association 1118, each device 302 isassociated with multiple bins 1116. Associating multiple bins 1116 toeach device 302 facilitates recoveries from failures of devices 302because the media data storage and sending responsibilities of a faileddevice 302 may be quickly spread among as many devices 302 as there arebins 1116 associated per device 302. Nevertheless, the number “b” ofbins 1116 of hash table 1114 is sufficiently small so that each device302 can afford to store a copy of hash table 1114 for quick reference.

Each device 302 that is associated 1118 with a given bin 1116 isresponsible for having in RAM the media data blocks 1006 that ultimatelymap 1112 (after hashing 1108) to the given bin 1116. If a requestedmedia data block 1006 is not in RAM, then the associated device 302retrieves it from mass storage of media data 310 (of FIG. 3).Regardless, the associated device 302 has a primary responsibility rolefor sending to clients 206 media data blocks 1006 that map 1112 to thegiven bin 1116. However, for highly popular media data, another device302 has a secondary responsibility role for sending media data blocks1006 that map 1112 to the given bin 1116. These primary and secondaryroles are described further below with reference to FIG. 13, anddetermining media data popularity is described below with reference toFIG. 12.

FIG. 12 illustrates an exemplary approach to a popularity determination1200 for media data. A given percentage of media data popularity doesnot necessarily equate to the same percentage of media data memory sizeor media data asset proportion. Exemplary popularity determination 1200determines media data blocks that amount to Z % of the popularity 1206.Popularity determination may alternatively be performed on a per-assetbasis, a per-track basis (e.g. an audio track in a particular languagevs. an audio track in another language), a per-sub-block basis, aper-byte basis, or at any other level of detail/memory size byapproximating the popularity of all media data at that level of detailas being equal.

Media data blocks 1006 are ranked from most popular to least popular inpopularity ranking 1202. As illustrated, media data blocks 1006 areranked from media data block 1006(1) to media data block 1006(2) tomedia data block 1006(3) to . . . media data block 1006(v). It should benoted that popularity ranking 1202 need not necessarily include mediadata blocks 1006 that are less popular than the Z % popularity level1206. It should be noted that media data blocks 1006 need not beconsidered individually if popularity determination 1200 is performed ona per-segment, per-asset, per-track, or any larger level of detail thanthe media data block level.

Each respective media data block 1006(1, 2, 3 . . . v) has beenrequested by a respective number of clients 1204(1, 2, 3 . . . v).Popularity ranking 1202 of media data blocks 1006 is effectuated withconsideration of the number of requesting clients 1204(1, 2, 3 . . . v).In other words, the number of requesting clients 1204(1) is greater thanthe number of requesting clients 1204(2), which is greater than thenumber of requesting clients 1204(3), and so forth.

As illustrated, the number of requesting clients 1204(1), 1204(2), and1204(3) total to Z % of the total number of clients 1208 to which mediadata is currently being disseminated. Hence, media data blocks 1006(1),1006(2), and 1006(3) comprise Z % of the popularity 1206 in the system.This Z % of the popularity 1206 is replicated as described further belowwith reference to FIG. 13.

In a described implementation, Z % of the popularity 1206 that isreplicated corresponds to 15%. It should be understood that 15% of therequests may be for 5% of the media data, in which case 5% of the mediadata is replicated in order to replicate 15% of the popularity.Popularity tracking can be performed on a per-media-data-block 1006basis, but other media data granularities may alternatively be trackedfor popularity and replication purposes. Regardless of the granularityat which popularity is tracked, popularities may be compared in terms ofpopularity per byte.

For the sake of clarity, popularity determination 1200 as illustrated inFIG. 12 conceals a further complicating subtlety. In a describedimplementation, popularity is determined on a per-equal-size-block (andthus a per-byte) basis in order to at least improve if not optimize RAMusage. In this described implementation, popularity in terms of requestsis tracked on a per-media-data-asset 1004 basis.

However, popularity ranking 1202 is also created responsive to aper-byte basis. For example, each number of clients 1204 (which istracked at the media data asset 1004 level and represents the number ofclients actively viewing the media data asset, or an average over recenttime of such a number) is divided by the total number of media datablocks 1006 in the corresponding media data asset 1004 and the result isfurther divided by the size in bytes of each media data block 1006.Using the resulting per-byte popularity value, popularity ranking 1202of media data blocks 1006 is created in popularity determination 1200.

To provide a clarifying example, a shorter media data asset 1004 has ahigher per-byte popularity than a longer media data asset 1004 that hasan equal number of current client requesters. If (i) the shorter mediadata asset 1004 has two media data blocks 1006 and the longer media dataasset 1004 has ten media data blocks 1006 of the same size and (ii) eachmedia data asset 1004 has 100 current requesting clients 206, then thereare more current requesting clients 206 per-media-data-block 1006 (andper-byte) for the shorter media data asset 1004 at any given instant.Causing the shorter media data asset 1004 and the media data blocks 1006thereof to have the higher popularity in this context results in a moreefficient usage of RAM-cached media data. An alternative approach thatprovides an equivalent per-byte popularity ranking result is to trackthe total number of bytes of a media data asset that are requested by aclient during a session since the most recent update of the popularitymetric. This total number of requested bytes of the media data asset isdivided by the total size of the media data asset in bytes, and thisquotient is then factored into the popularity metric periodically orwhen the session is over.

Using the per-byte popularity value, media data may be manipulated atthe media data block 1006 level responsive to current popularity. Suchmedia data manipulations, including replication and cached media dataoverflow purges, are described further below with reference to FIG. 13.

Although an implementation described above relies on popularity trackingat a per media-data-asset 1004 granularity, popularity may alternativelybe tracked at other levels of media data granularity. For example, mediadata popularity may be tracked at a per media-data-block 1006granularity level (in which case the popularity percentages areimplicitly on a per-media-data-block 1006 basis for popularity ranking1202 purposes). Furthermore, media data popularity may be tracked at aper media-data-block-set granularity level in which the set is apredetermined number of media data blocks 1006, such as ten media datablocks 1006.

FIG. 13 illustrates an exemplary hash table 1114 for locating mediadata, including replicated media data. Instead of each bin 1116 beingassociated with a single sender 304, each bin 1116 is associated 1118with a primary sender 304(P) and a secondary sender 304(S) as well asoptional tertiary and further senders. Generally, each bin 1116 isdivided into a primary sender role 1304 and a secondary sender role1302.

Primary senders 304(P) having primary sender roles 1304 are responsiblefor 100% of media data blocks 1006 that are mapped to a given bin 1116.Secondary senders 304(S) having secondary sender roles 1302 areresponsible for the top Z % of the popular media data blocks 1006 thatare mapped to a given bin 1116. Consequently, Z % of the popularity ofany given bin 1116 is replicated at the sender 304 having the secondarysender role 1302. As described above, replicating Z % of the popularityof the media data does not necessarily equate to having to replicate Z %of the size of the media data.

With specific reference to bin 1116(1), primary sender 304(P) isassociated therewith as having primary sender role 1304, and secondarysender 304(S) is associated with bin 1116(1) as having secondary senderrole 1302. Hence, media data blocks 1006 that map (e.g., throughhashing) to bin 1116(1) can be sent from primary sender 304(P). Thosemedia data blocks 1006 that map to bin 1116(1) and are part of the top Z% of current popularity in bin 1116(1) are replicated at and may also besent from secondary sender 304(S).

Secondary senders 304(S) may therefore be used to send media data blocks1006 of a given bin 1116 that are part of the top Z % of currentpopularity. This is especially useful for subsequent send requests 908(of FIG. 9) that are transmitted after a scheduler 306 learns that anoriginal send request 902 will not be honored by a primary sender 304(P)based on an initial threshold priority 904 that was received from theprimary sender 304(P).

Although not shown in FIG. 13, there may be additional sender roles(e.g., tertiary, quaternary, etc.) defined by hash table 1114 for mediadata mapped to bins 1116 thereof. Also, each additional sender role mayoptionally be for a different popularity percentage. For example, atertiary sender role may replicate the top 10% of the popularity. Insuch an example, three senders are responsible for the top 10% ofpopularity, two senders are responsible for the top 15% of popularity,and one sender is responsible for all 100% of popularity. A scheduler306 therefore has three sender 304 options if the send request is formedia data in the top 10% of popularity. Other numbers of senders 304and popularity replication percentages may alternatively be used.

Primary senders 304(P) are responsible for 100% of media data blocks1006 that are mapped to a given bin 1116. However, respective devices302 of respective primary senders 304(P) may not have sufficient RAM toconstantly store all media data blocks 1006 of a given bin 1116.Consequently, a primary sender 304(P) loads requested media datablock(s) 1006 from mass storage of media data 310 onto a respectivedevice 302 of primary sender 304(P) upon request from a scheduler 306.Latencies arising from accessing mass storage of media data 310 areameliorated by using a look-ahead scheme. This looking-ahead, along withrelated media-data-locking, is described further herein below in thesection entitled “Scheduling Scheme for Distributed Sending of MediaData”.

In a described implementation, when a “new” media data block 1006 isloaded by a sender 304, an “old” media data block 1006 is purged toprovide space in memory 404 of a device 302 of sender 304. Selection ofthe old media data block 1006 to be purged is made based on media datapopularity. For example, in a given bin 1116, a media data block 1006that is least popular is selected for purging and replacement. However,if the least popular media data block 1006 is locked due to a look-aheadprocedure, the next least popular media data block 1006 is consideredfor purging instead, and so on.

FIG. 14 illustrates an exemplary organization 1400 of media datasupporting information. Media data supporting information includesinformation that supports the presentation of audio/visual media datainformation. Organization 1400 includes media data stream schedule 1402and media data stream index 1406. Each media data stream schedule 1402is associated with a particular media data segment 1002, and eachcorresponding media data stream index 1406 (if also present) is likewiseassociated with the particular media data segment 1002.

In a described implementation, each media data stream schedule 1402includes information that is used by schedulers 306 to establish astreaming schedule that meets its obligations to an associateddestination client 206 that is assigned thereto. Schedulers 306 usemedia data stream schedules 1402 to ensure that media data408/1004/1006/1008 arrive in time to be processed and displayed withouta streaming failure. Each media data stream index 1406 indicatesmultiple points of random access to the associated media data segment1002. For example, media data stream index 1406 may include thelocations of I frames for certain types of encoded media data (e.g.,Moving Pictures Expert Group (MPEG)-encoded media data).

As illustrated, media data stream schedule 1402 includes multiplescheduling units 1404, including scheduling units 1404(q−1), 1404(q) . .. 1404(q+1). Media data stream index 1406 includes multiple indexingunits 1408, including indexing units 1408(r−1), 1408(r) . . . 1408(r+1).The number of scheduling units 1404 and indexing units 1408 do notnecessarily correspond to the number of media data blocks 1006 in theassociated media data segment 1002. Hence, a single scheduling unit 1404may provide supporting information for multiple media data blocks 1006.Although media data stream schedule 1402 is likely to be larger thanmedia data stream index 1406, their relative sizes are not necessarilyindicated in FIG. 14.

Each media data stream schedule 1402 may be stored in its entirety ateach device 302 for access by the scheduler 306 thereat. For athree-hour media data segment 1002, its associated media data streamschedule 1402 may be approximately 200 kB, which is manageable forrelatively small media data systems, with each scheduling unit 1404being about 1-8 kB. However, for improved scalability in a describedimplementation, scheduling units 1404 of each media data stream schedule1402 are distributed across multiple devices 302. When a scheduler 306wishes to access a particular scheduling unit 1404, it is located usinghash table 1114.

Scheduling units 1404 are therefore distributed across devices 302 usingthe same hashing function 1108 and hash table 1114 as is used fordistributing media data blocks 1006. Consequently, each media datastream schedule 1402 and individual scheduling unit 1404 thereof has anumber for determining an identification value for application tohashing function 1108. These numbers may be independent of theassociated media data segment 1002 and media data blocks 1006 thereofthat are supported by the scheduling units 1404, or they may be related.

For example, a number for a particular media data stream schedule 1402may be identical or similar to a media data segment number 1102 of anassociated particular media data segment 1002, with the numbering of thescheduling units 1404 continuing the numbering of the media data blocks1006. Alternatively, a media data segment 1002 may have an overarchingidentifying number with each of media data asset 1004, media data streamschedule 1402, and media data stream index 1406 (if included) havingsubstream identifiers that are based on the overarching identifyingnumber. Other numbering relationships may also be implemented.

More specifically, media data stream index 1406 is a file that listsseek points (random-access points) in an associated media data segment1002 and gives an offset into the associated media data segment 1002 forthat seek point. The entries of media data stream index 1406 are sortedby media time. Media data stream schedule 1402 is a more-detailed formof a media data stream index 1406. Media data stream schedules 1402include information for ascertaining a synchronization point for seekingto in an associated media data segment 1002, but they are generally morecumbersome than media data stream indexes 1406 for seeking because theyalso contain non-random-access locations.

Media data stream schedule 1402 addresses the frames or chunks of frames(in video or audio) of the associated media data segment 1002. It alsoincludes the decode time stamp (DTS) of each as well as the size of eachin bytes or packets. It is sorted by decode time stamp (DTS), includingpossibly reverse DTS.

The distinct nature of media data stream schedule 1402 enables thescheduling algorithm that is performed by schedulers 306 to be separatedfrom the audio/visual information of the associated media data segment1002. It should be understood that media data stream schedule 1402and/or media data stream index 1406 are exemplary approaches toorganizing media data information that supports the presentation ofaudio/visual media data information and that alternative organizationalimplementations may be employed instead.

Scheduling Scheme for Distributed Sending of Media Data

FIG. 15 illustrates exemplary media data characterizations 1500,including a look-ahead region 1502, that indicate differing media datamanipulation phases. A media data asset 1004 includes multiple mediadata blocks 1006(1), 1006(2) . . . 1006(7), 1006(8) . . . . Each mediadata block 1006 is associated with a sender 304 and stored at a device302 thereof via a hash striping media data distribution approach (e.g.,as described herein above with particular reference to FIGS. 11-13).

Although media data characterizations 1500 are illustrated as startingat media data block 1006(1), they may be generalized to start at anygiven media data block 1006. Media data asset 1004 is divided into acurrent block 1506, an alternative send request region 1504, and alook-ahead region 1502.

Current block 1506 corresponds to media data block 1006(1). An expandedview of media data block 1006(1) includes multiple media data sub-blocks1008(1), 1008(2), 1008(3) . . . 1008(s). As current block 1506, mediadata block 1006(1) is the media data block 1006 that is being processedby a remote client 206 (not shown in FIG. 15). It should be noted that,because of early send requests from alternative send request region1504, the media data block 1006 that is being processed by client 206may already be completely local to client 206 when it is time for client206 to process the first media data sub-block 1008(1).

On the other hand, if some media data sub-blocks 1008 are still beingtransmitted to client 206 when client 206 is processing media data block1006(1), one or more media data sub-blocks 1008 may be deadline data1514. Send requests 604 (of FIG. 6 et seq.) for such deadline data 1514are assigned the highest send priority to avoid a streaming failure.

If there is currently no deadline data 1514, and at least one sendrequest 604 for at least one non-deadline media data sub-block 1008 doesnot have a sufficiently high priority to exceed a threshold priority 704of the associated sender 304, a scheduler 306 may proceed to consideralternative send request region 1504 during a send negotiation pertimeslot 900 (of FIG. 9). (As described above with particular referenceto options evaluation 906 of FIG. 9, scheduler 306 may also considerincreasing an assigned send priority.) In order of consideration foralternative send request region 1504: (1) early data 1508, (2) laterearly data 1510, and (3) even later early data 1512 are considered for apotential send request 604. Later early data 1510, for example, isconsidered after early data 1508 because later early data 1510 isfurther into the future from a deadline time.

If a respective sender 304 that is associated with a respective mediadata block 1006(2), 1006(3), or 1006(4) has a threshold priority 704that is lower than what a send request 604, which stipulates anassociated respective media data block 1006, from the scheduler 306would have, then scheduler 306 formulates and transmits a send request604 to the respective sender 304 as part of a subsequent send request908 in an attempt to facilitate providing sender 304 with sufficientsend requests 604 (each having an associated priority) such that sender304 may fully and appropriately utilize its outgoing bandwidth 602 inthis timeslot by sending all the highest priority data that will fitwithin its outgoing bandwidth 602 to clients 206 in this timeslot.

Of media data blocks 1006(2, 3, and 4) of alternative send requestregion 1504, the respective media data block 1006 that has the earliestdata 1508/1510/1512 and a sufficiently high respective send priority isselected as the target for a send request 604. One or more media datasub-blocks 1008, including the next unsent media data sub-block 1008, ofthe selected media data block 1006 of alternative send request region1504 is stipulated by the send request 604 for sending to client 206. Inthis manner, a scheduler 306 is able to contribute to the saturation oflinks 314 (of FIGS. 3-6) from senders 304 while keeping a receptionbuffer at client 206 as full as possible (e.g., storing approximatelytwo media data blocks 1006 or about 10 seconds of media).

Look-ahead region 1502 can be instrumental in reducing latency.Implementing a look-ahead region 1502 with reserving and/or pre-loadingcan also enable a mass storage of media data 310 (of FIG. 3) to besomewhat if not significantly less robust and powerful than RAM.

Schedulers 306 maintain a reservation/pre-loading zone at look-aheadregion 1502 by attempting to ensure that a media data block 1006 thatwill be needed in the future will be present at the associated sender304, as determined by a hash striping media data distribution approach,for example. Schedulers 306 cause senders 304 to reserve, and ifnecessary pre-load, media data blocks 1006 in look-ahead region 1502using a look-ahead procedure (e.g., a look-ahead request, a look aheadoperation, etc.). These look-ahead reserving and/or pre-loading aspectsare described further below with reference to FIG. 16.

It should be noted that the illustrated collection of data and blocktypes, message regions, etc. is only an exemplary describedimplementation and that other configurations may alternatively beimplemented. Moreover, the illustrated collection effectively representsa snapshot in time for the exemplary configuration that actually changesdynamically. For example, the locations of early media data 1508, laterearly media data 1510, and even later early media data 1512 move alongmedia data asset 1004 as current block 1506 moves along media data asset1004 due to completing a sending of current block 1506 to a client 206.Also, alternative send request region 1504 may overlap look-ahead region1502. Additionally, either or both regions may extend from e.g. 2 to 16media data blocks 1006 from current block 1506; however, 2 to 8 mediadata blocks 1006 is likely a sufficient range.

FIG. 16 is an exemplary sequence diagram 1600 that involves a look-aheadprocedure between a scheduler 306 and a sender 304. Sender 304 includesa block usage counter (BUC) 1602 that effectively serves as a retentionlock on a corresponding media data block 1006 (not shown in FIG. 16). Asnoted above, a given sender 304, because of memory constraints of itsdevice 302, may not (and likely cannot) keep all of the media datablocks 1006 that hash 1108 to the given sender 304 in the RAM of itsdevice 302. Hence, at least some of these media data blocks 1006 arerotated between a memory 404 of device 302 and mass storage of mediadata 310. As noted above, media data blocks 1006 that are (i) less orleast popular and (ii) not locked may be purged.

Media data blocks 1006 are locked when one or more schedulers 306 havereserved them. This reservation may be accomplished using a look aheadrequest 1604 (e.g., a type of protocol message 504). In short, lookahead requests 1604 increment block usage counter 1602, and a media datablock 1006 cannot be purged while its corresponding block usage counter1602 is non-zero. There is therefore a corresponding block usage counter1602 for each media data block 1006. However, block usage counters mayalternatively be applied on a media data granularity level other thanmedia data blocks 1006.

An example of a look ahead procedure is described as follows: A blockusage counter 1602 equals zero (0) prior to any scheduler 306 requestinga look ahead on its corresponding media data block 1006. Hence, blockusage counter 1602(t=0) is shown as equaling zero at time=0. When ascheduler 306 is ready to look ahead at a particular media data block1006 (e.g., media data block 1006(7) of look-ahead region 1502), thescheduler 306 formulates and transmits a look ahead request 1604 thatidentifies the media data block 1006 of interest to a sender 304 that isresponsible for storing and sending that media data block 1006.

In response to receiving the look ahead request 1604, sender 304increments block usage counter 1602. Hence, block usage counter1602(t=t1) is shown as equaling one at time=t1. Furthermore, sender 304performs a look ahead operation 1606. Sender 304 checks/determineswhether the identified media data block 1006 is already in a RAM portionof memory 404. If so, sender 304 has completed look ahead operation 1606(once block usage counter 1602 has been incremented before, after, orduring performance of look ahead operation 1606).

If, on the other hand, the identified media data block 1006 is notalready in RAM portion of memory 404, sender 304 causes the identifiedmedia data block 1006 to be loaded into RAM from mass storage of mediadata 310 in addition to incrementing block usage counter 1602. Afterensuring that media data block 1006 is present in RAM (e.g., byverifying its presence and/or by loading it), sender 304 may optionallyinform scheduler 306 of the successful locking. Alternatively,schedulers 306 may be configured to send look ahead requests 1604sufficiently in advance so as to render this confirmation relativelyirrelevant.

After some period of time, scheduler 306 transmits a send request 1608(e.g., such as a send request 604) to sender 304 that stipulates theearlier-identified media data block 1006 by way of one or more mediadata sub-blocks 1008. Because the media data block 1006 is present andlocked by virtue of block usage counter 1602, sender 304 can send therequested one or more media data sub-blocks 1008 thereof to a designatedclient 206 (as indicated at 1610) without a delay attributable to mediadata loading.

After a retry period 1612 (e.g., after scheduler 306 is fairly sure ofsending success), scheduler 306 sends a look ahead cancel 1614 (e.g.,another type of protocol message 504) to sender 304. Look ahead cancel1614 identifies the media data block 1006 that is no longer in need ofreservation by scheduler 306. In response to look ahead cancel 1614,sender 304 decrements block usage counter 1602. In this example, noother schedulers have placed a lock on the identified media data block1006. Hence, block usage counter 1602(t=t2) is shown as equaling zero(0) again at time=t2, where t2 is later than t1. When block usagecounter 1602 is zero, the corresponding media data block 1006 isavailable to be considered for purging (e.g., swapping with a differentmedia data block 1006) in accordance with a given replacement policy(e.g., based on popularity).

Schedulers 306 have several options when/if choosing to lock a mediadata block 1006 on multiple senders 304 even though schedulers 306actually request media data sub-block 1008 sends from one sender 304 ata time; otherwise schedulers 306 do not actually have multiple senders304 as multiple sending options when the time arrives to request thesending of the media data sub-blocks 1008. Also, a scheduler 306 maychoose arbitrarily between senders 304 with which the scheduler 306 haslocked a media data block 1006 when doing originalmedia-data-sub-block-1008-level send requests 902 within that lockedmedia data block 1006.

Although subsequent send requests 908 may get restricted to specificsenders 304 due to initial and intermediate threshold priority 904 and910 constraints, original send requests 902 at the beginning of eachtimeslot have senders 304 that are chosen arbitrarily among those wherethe media data block 1006 is known to be locked by the requestingscheduler 306. In low-load situations where media data is not beingmoved around much, a scheduler 306 may elect to forgo locking all of thepossible media data block 1006 options, and in any case, the scheduler306 may limit itself to locking down only two (or another number) ofmedia data blocks 1006, even if there are more options available. Thiselection to not utilize all locking possibilities may be implemented,for example, to avoid locking down too much media data overall and toreduce protocol overhead.

It should be noted that some description herein (e.g., description thatis directed to popularity-dependent media data replication, to mediadata hash distribution (e.g., striping), to media data organization,etc.) is also beneficially applicable to and/or advantageouslyimplementable in conjunction with a central-collector type disseminationof media data.

Media data dissemination architecture 202, including individualdevices-302 thereof, may include a variety of processor-accessible mediaand/or may be in communication with such processor-accessible media. Theprocessor-accessible media may be any media that is accessible by acomputing or other (e.g., electronic) device. Hence, suchprocessor-accessible media may include both volatile and non-volatilemedia, both removable and non-removable media, both storage (e.g.,memory 404 and 310) and transmission (e.g., links 208, 210, 314, etc.)media, some combination thereof, and so forth. Any of the media maycomprise processor-executable instructions.

In fact, implementations for distributed sending of media data may bedescribed in the general context of processor-executable instructions.Generally, processor-executable instructions include routines, programs,protocols, objects, interfaces, components, data structures, etc. (e.g.,in the form of applications/modules) that perform and/or enableparticular tasks and/or implement particular abstract data types.Distributed sending of media data, as described in certainimplementations herein, may also be practiced in distributed processingenvironments where tasks are performed by remotely-linked processingdevices that are connected through a communications link and/or network.Especially but not exclusively in a distributed computing environment,processor-executable instructions may be located in separate storagemedia, executed by different processors, and/or propagated overtransmission media.

The devices, actions, aspects, features, procedures, components, etc. ofFIGS. 2-16 are illustrated in diagrams that are divided into multipleblocks. However, the order, interconnections, interrelationships,layout, etc. in which FIGS. 2-16 are described and/or shown is notintended to be construed as a limitation, and any number of the blockscan be modified, combined, rearranged, augmented, omitted, etc. in anymanner to implement one or more systems, methods, devices, procedures,media, apparatuses, servers, arrangements, etc. for distributed sendingof media data. Furthermore, although the description herein includesreferences to specific implementations, the illustrated and/or describedimplementations can be implemented in any suitable hardware, software,firmware, or combination thereof and using any suitable computingarchitecture(s), network element(s) and/or organization(s), videoencoding standard(s), multicast and unicast scheme(s), and so forth.

Although systems, media, devices, methods, procedures, apparatuses,techniques, schemes, approaches, procedures, arrangements, and otherimplementations have been described in language specific to structural,logical, algorithmic, and functional features and/or diagrams, it is tobe understood that the invention defined in the appended claims is notnecessarily limited to the specific features or diagrams described.Rather, the specific features and diagrams are disclosed as exemplaryforms of implementing the claimed invention.

1. One or more processor-accessible media comprisingprocessor-executable instructions that, when executed, direct at leastone device to perform actions comprising: transmitting a send requesthaving a priority; receiving a threshold priority; comparing thepriority of the send request to the threshold priority; and determiningif the send request is selected for sending responsive to the comparing.2. The one or more processor-accessible media as recited in claim 1,wherein the action of transmitting a send request comprises an actionof: transmitting the send request to a sender that is capable ofimplementing the send request by sending media data stipulated in thesend request to a destination client designated in the send request. 3.The one or more processor-accessible media as recited in claim 2,wherein the action of receiving a threshold priority comprises an actionof: receiving the threshold priority from the sender that is capable ofimplementing the send request, the threshold priority representing asend request cutoff that indicates a least important priority for sendrequests selected for sending by the sender.
 4. The one or moreprocessor-accessible media as recited in claim 1, wherein the action oftransmitting a send request comprises an action of: transmitting thesend request which is associated with a unique priority; wherein thesend request stipulates media data, and the unique priority is dependentat least in part on whether the stipulated media data comprises deadlinemedia data versus early media data, with deadline media data having arelatively higher priority.
 5. The one or more processor-accessiblemedia as recited in claim 1, wherein the action of transmitting a sendrequest comprises an action of: transmitting the send request which isassociated with a unique priority; wherein the send request stipulatesmedia data, and the unique priority is dependent at least in part onwhether the stipulated media data is associated with fewer capablesenders versus many capable senders, with association with fewer capablesenders having a relatively higher priority.
 6. The one or moreprocessor-accessible media as recited in claim 1, wherein the actions ofcomparing and determining comprise actions of: ascertaining if thepriority of the send request is more important than the thresholdpriority; and if so, determining that the send request is selected forsending of media data as stipulated by the send request.
 7. The one ormore processor-accessible media as recited in claim 6, comprising theprocessor-executable instructions that, when executed, direct the atleast one device to perform a further action comprising: if the priorityof the send request is not ascertained to be more important than thethreshold priority, determining that the send request is not selectedfor sending of media data as stipulated by the send request.
 8. The oneor more processor-accessible media as recited in claim 1, wherein theaction of transmitting a send request comprises an action oftransmitting the send request to a sender that is capable ofimplementing the send request; and wherein the processor-executableinstructions, when executed, direct the at least one device to performfurther actions comprising: if the send request is not determined to beselected for sending responsive to the comparing, determining if thepriority of the send request is more important than at least one otherthreshold priority received from at least one other sender that iscapable of implementing the send request; and if so, transmitting thesend request to the at least one other sender that is capable ofimplementing the send request.
 9. The one or more processor-accessiblemedia as recited in claim 1, wherein at least a portion of theprocessor-executable instructions comprise at least part of softwareimplementing a scheduler that is adapted to schedule sending of mediadata portions of a media data stream to clients.
 10. One or moreprocessor-accessible media comprising processor-executable instructionsthat, when executed, direct at least one device to perform actionscomprising: receiving a plurality of send requests, each respective sendrequest of the plurality of send requests having a respective priority;ascertaining a threshold priority based on the respective priorities ofthe respective send requests and responsive to a send bandwidth; andbroadcasting the threshold priority.
 11. The one or moreprocessor-accessible media as recited in claim 10, wherein the action ofreceiving a plurality of send requests comprises an action of: receivingthe plurality of send requests from a plurality of schedulers thatformulated each send request of the plurality of send requests toinclude a stipulated media data portion, to designate a destinationclient, and to have a unique priority.
 12. The one or moreprocessor-accessible media as recited in claim 10, comprising theprocessor-executable instructions that, when executed, direct the atleast one device to perform a further action comprising: ranking theplurality of send requests according to respective priorities ofrespective send requests of the plurality of send requests.
 13. The oneor more processor-accessible media as recited in claim 12, wherein theaction of ascertaining a threshold priority comprises an action of:detecting a particular send request of the plurality of send requeststhat corresponds to a send request cutoff using the ranked plurality ofsend requests and the send bandwidth.
 14. The one or moreprocessor-accessible media as recited in claim 10, wherein the action ofascertaining a threshold priority comprises an action of: accumulatingbandwidth consumption of the plurality of send requests from a sendrequest that is most important to those that are less important inaccordance with the respective priorities thereof.
 15. The one or moreprocessor-accessible media as recited in claim 14, wherein the action ofascertaining a threshold priority comprises a further action of:determining a last send request at which the accumulated bandwidthconsumption is less than or equal to the send bandwidth.
 16. The one ormore processor-accessible media as recited in claim 15, wherein theaction of ascertaining a threshold priority comprises a further actionof: setting the threshold priority to a priority corresponding to thelast send request at which the accumulated bandwidth consumption is lessthan or equal to the send bandwidth.
 17. The one or moreprocessor-accessible media as recited in claim 10, comprising theprocessor-executable instructions that, when executed, direct the atleast one device to perform further actions comprising: discarding sendrequests of the plurality of send requests that do not fit within thesend bandwidth; and partially discarding and partially retaining a sendrequest of the plurality of send requests that is only partially withinthe send bandwidth.
 18. The one or more processor-accessible media asrecited in claim 10, wherein the action of broadcasting the thresholdpriority comprises at least one action of: unicasting the thresholdpriority to a plurality of schedulers; or multicasting the thresholdpriority to the plurality of schedulers.
 19. The one or moreprocessor-accessible media as recited in claim 10, wherein at least aportion of the processor-executable instructions comprise at least partof software implementing a sender that is adapted to send to clientsmedia data portions with which the sender is associated responsive tothe plurality of send requests.
 20. An arrangement for prioritizeddistributed sending of media data, the arrangement comprising: schedulermeans for requesting sending of media data via send requests having sendpriorities; the scheduler means including determination means fordetermining if the send requests have been selected for sending usingthe send priorities and a threshold priority; and sender means forsending media data portions in response to the send requests inaccordance with the send priorities; the sender means includingascertainment means for ascertaining the threshold priority based on thesend priorities of the send requests and responsive to a send bandwidthof the sender means.
 21. The arrangement as recited in claim 20, whereinthe scheduler means further comprises: formulation means for originallyformulating the send requests to include designated destination clients,stipulated media data portions, and the send priorities.
 22. Thearrangement as recited in claim 20, wherein the scheduler means furthercomprises: transmission means for transmitting each send request of thesend requests to a sender means of a plurality of sender means, eachsender means of the plurality of sender means having at least one sendrequest being transmitted thereto in dependence on which media dataportions are associated therewith.
 23. The arrangement as recited inclaim 20, wherein the determination means further comprises: comparisonmeans for comparing the send priorities to the threshold priority. 24.The arrangement as recited in claim 20, wherein the scheduler meansfurther comprises: checking means for checking whether another thresholdpriority is lower than the send priorities of send requests that are notselected for sending.
 25. The arrangement as recited in claim 20,further comprising: a plurality of scheduler means and a plurality ofsender means that are exchanging send requests and threshold priorities.26. The arrangement as recited in claim 20, wherein the sender meansfurther comprises: ranking means for ranking the send requests from mostimportant to least important according to the send priorities.
 27. Thearrangement as recited in claim 20, wherein the arrangement comprises atleast one of (i) one or more processor-accessible media or (ii) at leastone device.
 28. A system that is configured to perform actionscomprising: transmitting a send request associated with a send priorityfrom a scheduler to a sender; ascertaining a threshold priority at thesender based on a ranking of send requests and responsive to a sendbandwidth of the sender, the ranking of send requests including the sendrequest from the scheduler; transmitting the threshold priority to thescheduler; and determining at the scheduler if the send request is beinghonored by the sender using the threshold priority and the send priorityassociated with the send request.
 29. The system as recited in claim 28,wherein the system is configured to perform further actions comprising:ranking the send requests in order of respective priorities ofrespective send requests of the send requests to produce the ranking ofsend requests; and detecting a send request cutoff corresponding to asend request (i) that fits within the send bandwidth of the sender and(ii) that is associated with a lowest send priority of all send requeststhat fit within the send bandwidth of the sender.
 30. The system asrecited in claim 28, wherein the system comprises a plurality ofdevices; and wherein (i) the scheduler and the sender are functioning ona single device of the plurality of devices or (ii) the scheduler isfunctioning on a first device of the plurality of devices, and thesender is functioning on a second device of the plurality of devices.31. The system as recited in claim 28, wherein the system is configuredto perform further actions comprising: receiving the send request at thesender from the scheduler; and transmitting, in response to thereceiving, an acknowledgment from the sender to the scheduler if thesend priority associated with the send request is more important than acurrent threshold priority of the sender.
 32. The system as recited inclaim 28, wherein the action of determining comprises: determining thatthe send request is being honored by the sender if (i) the send priorityassociated with the send request is as important or more important thanthe threshold priority of the sender, (ii) an acknowledgment is receivedat the scheduler from the sender in response to the transmitting of thesend request, and (iii) the threshold priority is marked by the senderas being final.
 33. The system as recited in claim 28, wherein thesystem is configured to perform further actions comprising: if the sendrequest is not determined to be honored by the sender, then: checking,at the scheduler, if another sender that is capable of sending mediadata as stipulated in the send request has transmitted another thresholdpriority that is less important than the send priority associated withthe send request; and if so, transmitting a subsequent send requestassociated with the send priority from the scheduler to the othersender.
 34. The system as recited in claim 28, wherein the system isconfigured to perform further actions comprising: if the send request isnot determined to be honored by the sender, then: checking, at thescheduler, if another sender has transmitted another threshold prioritythat is less important than another send priority that is assignable tothe send request by the scheduler; and if so, transmitting from thescheduler to the other sender a subsequent send request associated withthe other send priority as assigned by the scheduler.
 35. One or moreprocessor-accessible media comprising processor-executable instructionsthat, when executed, cause a system to create schedulers that arecapable of formulating send requests having send priorities in order toprovide media data streams to clients; to establish senders that arerespectively associated with respective media data portions and adaptedto ascertain respective threshold priorities each time slot based onrespective prioritized rankings of the respective send requests receivedat each sender from the schedulers and responsive to respective sendbandwidths of each respective sender; and to enable the schedulers todetermine if send requests are being honored by respective senders usingthe respective ascertained threshold priorities thereof.
 36. The one ormore processor-accessible media as recited in claim 35, comprising theprocessor-executable instructions wherein each respective send bandwidthcomprises a percentage, which is less than 100 percent, of a respectivemaximum outgoing bandwidth that is available at each respective senderof the established senders.
 37. The one or more processor-accessiblemedia as recited in claim 35, comprising the processor-executableinstructions wherein the schedulers are adapted to transmit subsequentsend requests to the senders in response to the ascertained thresholdpriorities.
 38. The one or more processor-accessible media as recited inclaim 37, comprising the processor-executable instructions whereinrespective senders of the senders are further adapted to ascertainrespective intermediate threshold priorities within each time slot basedon respective prioritized rankings of the subsequent send requestsreceived at each sender from the schedulers and responsive to therespective send bandwidth of each respective sender.
 39. The one or moreprocessor-accessible media as recited in claim 37, comprising theprocessor-executable instructions wherein the subsequent send requestsare at least one of (i) transmitted to different senders as compared tooriginal send requests and/or (ii) assigned a more important sendpriority as compared to the original send requests.
 40. The one or moreprocessor-accessible media as recited in claim 35, comprising theprocessor-executable instructions wherein each respective scheduler usesno more than a particular percentage, with the particular percentagebeing less than 100 percent, of a respective allotted portion of anaggregate send bandwidth of the senders that has been allotted to eachrespective scheduler when allocating media data sends to ensure futureguaranteed media data delivery deadlines.